Skip to content

[codex] replace backfill chunking with smart planner#107

Draft
KeKs0r wants to merge 5 commits intomainfrom
new-smart-chunking
Draft

[codex] replace backfill chunking with smart planner#107
KeKs0r wants to merge 5 commits intomainfrom
new-smart-chunking

Conversation

@KeKs0r
Copy link
Copy Markdown
Contributor

@KeKs0r KeKs0r commented Apr 1, 2026

Summary

  • replace the old backfill partition/range chunking path with the new smart recursive chunk planner
  • expose planner/runtime internals from @chkit/plugin-backfill/sdk instead of the package root
  • add fixture-driven integration coverage for each chunking strategy and mixed multi-dimension scenarios
  • fix review findings around degenerate equal-width slices, sort-key expression parsing, temporal slice bounds, and package export validation

Why

The previous backfill planner only handled coarse partition splitting with a single sort-key range fallback. The new smart planner supports string-prefix, temporal, quantile, and equal-width strategies, including secondary-dimension handoff for hot keys, which matches the new prototype behavior and gives much better confidence in expected chunk layouts.

Impact

Backfill planning now produces smarter chunk boundaries for skewed data and multi-column sort keys.

Public API note: planner/runtime internals are no longer exported from @chkit/plugin-backfill. They now live under @chkit/plugin-backfill/sdk.

Root Cause

The older planner could only split by partition and a single coarse sort-key range, which broke down on hot keys and skewed distributions. During review, a few edge cases also surfaced in the first port: collapsed equal-width boundaries, brittle sorting_key parsing for expression-based sort keys, and temporal refinement extending past parent slice bounds.

Validation

  • bun test /Users/marc/Workspace/chkit/packages/plugin-backfill/src
  • bun run --cwd /Users/marc/Workspace/chkit/packages/plugin-backfill typecheck
  • bun verify

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant